KEP-24: Promote Apparmor to GA #3298

mccormickt · 2022-05-07T13:17:03Z

KEP-24: Promote Apparmor to GA

New updated KEP outlining requirement for Apparmor to move to GA.

Issue link: AppArmor support #24

Supersedes #1444

Signed-off-by: Sascha Grunert <sgrunert@suse.com> Signed-off-by: Sascha Grunert <sgrunert@redhat.com>

…ancements

k8s-ci-robot · 2022-05-07T13:17:10Z

Welcome @jan0ski!

It looks like this is your first PR to kubernetes/enhancements 🎉. Please refer to our pull request process documentation to help your PR have a smooth ride to approval.

You will be prompted by a bot to use commands during the review process. Do not be afraid to follow the prompts! It is okay to experiment. Here is the bot commands documentation.

You can also check if kubernetes/enhancements has its own contribution guidelines.

You may want to refer to our testing guide if you run into trouble with your tests not passing.

If you are having difficulty getting your pull request seen, please follow the recommended escalation practices. Also, for tips and tricks in the contribution process you may want to read the Kubernetes contributor cheat sheet. We want to make sure your contribution gets all the attention it needs!

Thank you, and welcome to Kubernetes. 😃

k8s-ci-robot · 2022-05-07T13:17:11Z

Hi @jan0ski. Thanks for your PR.

I'm waiting for a kubernetes member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

kikisdeliveryservice

A couple of general comments, but since this is based on a kep draft from Sacha from a year ago, some changes have occurred in the KEP template. I've marked the two sections that definitely need to pull in the new template (PRR and Test Plan), but suggest that you go through and ensure you're using the current version for all sections :)

keps/sig-node/24-apparmor-ga/README.md

dchen1107 · 2022-06-21T17:48:01Z

/assign @tallclair

sftim · 2022-07-17T11:31:58Z

Before we graduate this, can we check whether https://kubernetes.io/docs/tutorials/security/apparmor/ is current? That page talks a lot about requiring Kubernetes v1.4 or later, so my guess is that it does need an update.

sftim

The path keps/sig-node/24-apparmor-ga looks wrong. We usually don't put -ga suffixes on short names for KEPs; the intention to stabilize is implicit.

mccormickt · 2023-02-03T21:25:39Z

Looks like there are still some PRR questions open.
...
If you have time to address the open debuggability concerns that works too.

@tallclair can you point these concerns out for me? I might be missing context, and had assumed PRR questions were addressed already. Happy to try and answer whatever I can.

deads2k · 2023-02-08T16:19:11Z

Back in october the PRR was approved #3298 (comment)

There was a new question added to scalability: https://github.com/kubernetes/enhancements/blob/master/keps/NNNN-kep-template/README.md#scalability , but that should be easy to update.

SergeyKanzhelev · 2023-02-08T23:48:30Z

keps/sig-node/24-apparmor/README.md

+##### e2e tests
+
+AppArmor already has [e2e tests][https://github.com/kubernetes/kubernetes/blob/2f6c4f5eab85d3f15cd80d21f4a0c353a8ceb10b/test/e2e_node/apparmor_test.go],
+but the tests are guarded by the `[Feature:AppArmor]` tag and not run in the


it is running in some jobs: https://storage.googleapis.com/k8s-triage/index.html?test=AppArmor

SergeyKanzhelev · 2023-02-08T23:53:19Z

keps/sig-node/24-apparmor/README.md

+The AppArmor fields on a pod are immutable, which also applies to the
+[annotation](https://github.com/kubernetes/kubernetes/blob/b46612a74224b0871a97dae819f5fb3a1763d0b9/pkg/apis/core/validation/validation.go#L177-L182).
+
+When an [Ephemeral Container](20190212-ephemeral-containers.md) is added, it


ephemeral containers are GA now: #277

The tests section doesn't mention e2e tests validating ephemeral containers behavior. Should this be added?

keps/sig-node/24-apparmor/kep.yaml

SergeyKanzhelev · 2023-02-08T23:56:49Z

All the feedback left on this KEP can be addressed by adding action items into the Graduation Criteria for GA. There is no absolutely blocking comments, mostly work items

k8s-ci-robot · 2023-02-10T01:31:37Z

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: deads2k, jan0ski, tallclair
Once this PR has been reviewed and has the lgtm label, please ask for approval from derekwaynecarr. For more information see the Kubernetes Code Review Process.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

~~keps/prod-readiness/OWNERS~~ [deads2k]
keps/sig-node/OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

SergeyKanzhelev · 2023-02-23T20:38:24Z

keps/sig-node/24-apparmor/README.md

+available to Localhost profiles. This prevents profiles meant for other system
+daemons to be utilized by Kubernetes and will be configurable by the kubelet.
+
+###### Updating AppArmor profiles


Would be nice to cover this behavior as well: kubernetes/kubernetes#116003

k8s-triage-robot · 2023-05-24T20:46:09Z

The Kubernetes project currently lacks enough contributors to adequately respond to all PRs.

This bot triages PRs according to the following rules:

After 90d of inactivity, lifecycle/stale is applied
After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
After 30d of inactivity since lifecycle/rotten was applied, the PR is closed

You can:

Mark this PR as fresh with /remove-lifecycle stale
Close this PR with /close
Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

k8s-triage-robot · 2023-06-23T21:45:56Z

The Kubernetes project currently lacks enough active contributors to adequately respond to all PRs.

This bot triages PRs according to the following rules:

After 90d of inactivity, lifecycle/stale is applied
After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
After 30d of inactivity since lifecycle/rotten was applied, the PR is closed

You can:

Mark this PR as fresh with /remove-lifecycle rotten
Close this PR with /close
Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle rotten

pjbgf · 2023-07-04T09:11:36Z

/remove-lifecycle rotten

tallclair · 2024-01-12T19:14:46Z

@mccormickt I'd like to drive this forward in v1.30. Are you still open to working on it, or should I take over this PR?

tallclair · 2024-01-12T19:38:43Z

keps/sig-node/24-apparmor/README.md

+| 2) Using custom or `runtime/default` profile that restricts actions a container is trying to make. | Pod created          | The outcome is workload and AppArmor dependent. In this scenario containers may 1) fail to start, 2) misbehave or 3) log violations.                  |
+| 3) Using custom profile that does not exist on the node.                                           | Pod created          | Containers fail to start. Retry respecting RestartPolicy and back-off delay.                                                                          |
+| 4) Using an unsupported runtime profile (i.e. `runtime/default-audit`).                            | Pod **not** created. | N/A                                                                                                                                                   |
+| 5) Using localhost profile with invalid name                                                       | Pod **not** created. | N/A                                                                                                                                                   |


There is not currently any validation of the localhost profile name. We should add field validation, but need to handle annotation validation carefully.

tallclair · 2024-01-12T19:40:33Z

keps/sig-node/24-apparmor/README.md

+object, events, or a warning as described in [KEP
+#1693](/keps/sig-api-machinery/1693-warnings).
+
+#### PodSecurityPolicy Support


Needs to be updated for PodSecurity admission

tallclair · 2024-01-12T19:41:29Z

keps/sig-node/24-apparmor/README.md

+| -------------------------------------------------------------------------------------------------- | -------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------- |
+| 1) Using localhost or `runtime/default` profile when container runtime does not support AppArmor.     | Pod created          | The outcome is container runtime dependent. In this scenario containers may 1) fail to start or 2) run normally without having its policies enforced. |
+| 2) Using custom or `runtime/default` profile that restricts actions a container is trying to make. | Pod created          | The outcome is workload and AppArmor dependent. In this scenario containers may 1) fail to start, 2) misbehave or 3) log violations.                  |
+| 3) Using custom profile that does not exist on the node.                                           | Pod created          | Containers fail to start. Retry respecting RestartPolicy and back-off delay.                                                                          |


Suggested change

| 3) Using custom profile that does not exist on the node. | Pod created | Containers fail to start. Retry respecting RestartPolicy and back-off delay. |

| 3) Using localhost profile that does not exist on the node. | Pod created | Containers fail to start. Retry respecting RestartPolicy and back-off delay. |

tallclair · 2024-01-12T19:42:19Z

keps/sig-node/24-apparmor/README.md

+| -------------------------------------------------------------------------------------------------- | -------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------- |
+| 1) Using localhost or `runtime/default` profile when container runtime does not support AppArmor.     | Pod created          | The outcome is container runtime dependent. In this scenario containers may 1) fail to start or 2) run normally without having its policies enforced. |
+| 2) Using custom or `runtime/default` profile that restricts actions a container is trying to make. | Pod created          | The outcome is workload and AppArmor dependent. In this scenario containers may 1) fail to start, 2) misbehave or 3) log violations.                  |
+| 3) Using custom profile that does not exist on the node.                                           | Pod created          | Containers fail to start. Retry respecting RestartPolicy and back-off delay.                                                                          |


Is this outcome managed by the Kubelet or the container runtime?

How is the error surfaced to the user?

tallclair · 2024-01-12T19:48:39Z

keps/sig-node/24-apparmor/README.md

+If a Pod with a container specifies an AppArmor profile by field/annotation,
+then the container will only apply the Pod level field/annotation if none
+are set on the container level.


Suggested change

If a Pod with a container specifies an AppArmor profile by field/annotation,

then the container will only apply the Pod level field/annotation if none

are set on the container level.

Container-level AppArmor profiles override anything set at the pod-level.

tallclair · 2024-01-12T19:48:56Z

keps/sig-node/24-apparmor/README.md

+To raise awareness of annotation usage (in case of old automation), a warning
+mechanism will be used to highlight that support will be dropped in v1.30.
+The mechanisms being considered are audit annotations, annotations on the
+object, events, or a warning as described in [KEP
+#1693](/keps/sig-api-machinery/1693-warnings).


Needs to be updated.

tallclair · 2024-01-12T19:49:35Z

keps/sig-node/24-apparmor/README.md

+To raise awareness of existing controllers using the AppArmor annotations that
+need to be migrated, a warning mechanism will be used to highlight that support
+will be dropped in v1.30.


Needs update.

tallclair · 2024-01-12T19:49:57Z

keps/sig-node/24-apparmor/README.md

+#### PodTemplates
+
+PodTemplates (e.g. ReplaceSets, Deployments, StatefulSets, etc.) will be
+ignored. The field/annotation resolution will happen on template instantiation.


What are the implications for PodSecurity admission?

tallclair · 2024-01-12T19:50:34Z

keps/sig-node/24-apparmor/README.md

+Since
+[we support](https://kubernetes.io/docs/setup/release/version-skew-policy/) up
+to 2 minor releases of version skew between the master and node, annotations
+must continue to be supported and backfilled for at least 2 versions passed the
+initial implementation. If this feature is implemented in v1.27, I propose v1.30 as a
+target for removal of the old behavior. Specifically, annotation support will be removed
+in the kubelet after this period, and fields will no longer be copied to annotations for older kubelet
+versions. However, annotations submitted to the API server will continue to be copied to fields at the
+kubelet indefinitely, as was done with Seccomp.


Timeline needs update

mccormickt · 2024-01-14T23:41:30Z

@mccormickt I'd like to drive this forward in v1.30. Are you still open to working on it, or should I take over this PR?

Thanks @tallclair, please feel free to take this over. Apologies for the lack of recent communication, life has gotten in the way of progress here as of late. Let me know if I can help clarify any gaps.

tallclair · 2024-01-19T00:18:06Z

Closing in favor of #4417.

saschagrunert and others added 4 commits May 19, 2021 09:30

Add AppArmor GA KEP

685e057

Signed-off-by: Sascha Grunert <sgrunert@suse.com> Signed-off-by: Sascha Grunert <sgrunert@redhat.com>

Merge branch 'apparmor-ga' of github.com:saschagrunert/kubernetes-enh…

c03ab9c

…ancements

Remove PSP references

f3489ba

Update kep metadata

3ae85d2

k8s-ci-robot added do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. labels May 7, 2022

k8s-ci-robot added the needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. label May 7, 2022

k8s-ci-robot added the size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. label May 7, 2022

k8s-ci-robot requested a review from dchen1107 May 7, 2022 13:17

k8s-ci-robot added the kind/kep Categorizes KEP tracking issues and PRs modifying the KEP directory label May 7, 2022

k8s-ci-robot requested a review from derekwaynecarr May 7, 2022 13:17

k8s-ci-robot added the sig/node Categorizes an issue or PR as relevant to SIG Node. label May 7, 2022

kikisdeliveryservice reviewed May 10, 2022

View reviewed changes

mccormickt added 3 commits May 16, 2022 22:13

Update KEP template and metadata

8aef2f2

Add initial design and beta release to implementation history

24ca438

Update stable target to 1.25

de1d3bd

mccormickt marked this pull request as ready for review May 17, 2022 02:44

k8s-ci-robot removed the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label May 17, 2022

mccormickt mentioned this pull request Jun 14, 2022

Add AppArmor GA fields and validation kubernetes/kubernetes#110552

Closed

k8s-ci-robot assigned tallclair Jun 21, 2022

mccormickt added 2 commits June 22, 2022 21:13

Change target to 1.26

fca578e

Change target to 1.26

db15333

kikisdeliveryservice mentioned this pull request Jul 7, 2022

KEP for promoting AppArmor to GA #1444

Closed

sftim reviewed Jul 17, 2022

View reviewed changes

dchen1107 added this to the v1.26 milestone Jul 28, 2022

SergeyKanzhelev reviewed Feb 8, 2023

View reviewed changes

keps/sig-node/24-apparmor/kep.yaml Show resolved Hide resolved

SergeyKanzhelev reviewed Feb 8, 2023

View reviewed changes

keps/sig-node/24-apparmor/kep.yaml Show resolved Hide resolved

add dchen1107 as an approver/reviewer

fddcbb9

SergeyKanzhelev reviewed Feb 23, 2023

View reviewed changes

SergeyKanzhelev mentioned this pull request Mar 7, 2023

AppArmor validator: Use up to date host info kubernetes/kubernetes#116003

Closed

k8s-ci-robot added the lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. label May 24, 2023

pjbgf mentioned this pull request May 30, 2023

remove deprecated apparmorProfile and redundant type conversion kubernetes/kubernetes#118314

Closed

k8s-ci-robot added lifecycle/rotten Denotes an issue or PR that has aged beyond stale and will be auto-closed. and removed lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. labels Jun 23, 2023

k8s-ci-robot removed the lifecycle/rotten Denotes an issue or PR that has aged beyond stale and will be auto-closed. label Jul 4, 2023

tallclair added the triage/accepted Indicates an issue or PR is ready to be actively worked on. label Aug 21, 2023

tallclair modified the milestones: v1.27, v1.30 Jan 12, 2024

tallclair reviewed Jan 12, 2024

View reviewed changes

tallclair mentioned this pull request Jan 19, 2024

KEP-24: Promote AppArmor to GA #4417

Merged

tallclair closed this Jan 19, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

KEP-24: Promote Apparmor to GA #3298

KEP-24: Promote Apparmor to GA #3298

mccormickt commented May 7, 2022

k8s-ci-robot commented May 7, 2022

k8s-ci-robot commented May 7, 2022

kikisdeliveryservice left a comment

dchen1107 commented Jun 21, 2022

sftim commented Jul 17, 2022

sftim left a comment

mccormickt commented Feb 3, 2023

deads2k commented Feb 8, 2023

SergeyKanzhelev Feb 8, 2023

SergeyKanzhelev Feb 8, 2023

SergeyKanzhelev commented Feb 8, 2023

k8s-ci-robot commented Feb 10, 2023

SergeyKanzhelev Feb 23, 2023

k8s-triage-robot commented May 24, 2023

k8s-triage-robot commented Jun 23, 2023

pjbgf commented Jul 4, 2023

tallclair commented Jan 12, 2024

tallclair Jan 12, 2024

tallclair Jan 12, 2024

tallclair Jan 12, 2024

tallclair Jan 12, 2024

tallclair Jan 12, 2024

tallclair Jan 12, 2024

tallclair Jan 12, 2024

tallclair Jan 12, 2024

tallclair Jan 12, 2024

tallclair Jan 12, 2024

mccormickt commented Jan 14, 2024 •

edited

Loading

tallclair commented Jan 19, 2024

	\| 3) Using custom profile that does not exist on the node. \| Pod created \| Containers fail to start. Retry respecting RestartPolicy and back-off delay. \|
	\| 3) Using localhost profile that does not exist on the node. \| Pod created \| Containers fail to start. Retry respecting RestartPolicy and back-off delay. \|

KEP-24: Promote Apparmor to GA #3298

KEP-24: Promote Apparmor to GA #3298

Conversation

mccormickt commented May 7, 2022

k8s-ci-robot commented May 7, 2022

k8s-ci-robot commented May 7, 2022

kikisdeliveryservice left a comment

Choose a reason for hiding this comment

dchen1107 commented Jun 21, 2022

sftim commented Jul 17, 2022

sftim left a comment

Choose a reason for hiding this comment

mccormickt commented Feb 3, 2023

deads2k commented Feb 8, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

SergeyKanzhelev commented Feb 8, 2023

k8s-ci-robot commented Feb 10, 2023

Choose a reason for hiding this comment

k8s-triage-robot commented May 24, 2023

k8s-triage-robot commented Jun 23, 2023

pjbgf commented Jul 4, 2023

tallclair commented Jan 12, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mccormickt commented Jan 14, 2024 • edited Loading

tallclair commented Jan 19, 2024

mccormickt commented Jan 14, 2024 •

edited

Loading